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ABSTBACT 

A three-year project developed a coapiiter*based 
systea of speech* training aids for the deaf* foranlated 
speech-training procedures that ntilize the systea, and evalnated 
these procedures. The first year was devoted to developaent of the 
systea; in succeeding years it was eaployed in a reaedial 
speech*training prograa at Clarke School for the Deaf. The systea and 
its use at Clarke School are described, along with results of an 
atteapt to evaluate its effectiveness. ( Author /SK) 
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Abstract 

This is tho final report of a three-year project to 
develop a computt^r-based system of spcoch-training aids for 
the deaf, to formulate speech-training procedures that utilise 
the system, and to evaluate these procedures. The first year 
of the project was devoted to the development of the system. 
During the second and third years Lhe system was used in a 
remedial speech-training program at the Clarke School for the 
Deaf. This report, along with several other to which it refers, 
describes the system, its use at the Clarke School, and the 
results of an attempt to evaluate the ef f ec ; : voness of the 
approach to speech training that it represent... 
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1. INTRODUCTION 

This if? tho firml report of. a project to develop and evaluate 
a system of conputcr-based specrh-traininq aids for the deaf under 
Contract No. OT:c-01-71-4G70. Tho project was iniLiated on 1 June 
1971 and comjilotod on 14 September 1974 . 

To our knowledge, thiB project reprenentc tho first attcnipt 
to use a computer to qonerate reai-time visu.il displays of speech 
properties for the purpose of facilitating the teaching of spoocli 
to deaf children. From the beginning of the project wo have felt 
that the best way to proceed would be to develop a prototype 
system that could bo easily modified, to get it quickly into the 
hands of users, and then to develop the system's capability oiiJ 
further in accordance with the insights gained as a result of 
trying to use it. Accordingly, the basic system was configured 
and the initial programming done during tho first year of tho 
project, and then the system v;aE installed at the Clarke School 
for the Deaf where it has been used on an almost-daily basis for 
two years. The system has continued to evolve, and many of the 
characteristics of the displays that are currently operational 
are the results of suggestions made by the users. 

A particularly gratifying aspect of the project has been 
the smoothness of the interaction betv;een the researchers and 
the teachers involved. Frequent and regularly scheduled meetings 
between the BBN and the Clarke Scliool members of the project team 
have facilitated the exchange of ideas and assured a continuing 
commonality of purpose. Many writers have pleaded for closer 
collaborations between researclicrs and teachers on the problem 
of developing procedures for teaching speech to the deaf. Our 
experience convinces us of the reasonableness of this plea. Each 
group has much to learn from the other, and the problem is cer- 
tainly difficult enough to require the best efforts of both. 
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2. CONTKNT AND ORGANIZATION OP Till; REPORT 

Much of tho work that has been done under this project has 
been documented in a scries of proqress reports and technical papers. 
The proqress roi>orts have been issued and submitted to BT'H on a 
quarterly basis. These reports contain many details of the dcsifjn 
and use of the systeii^, and provide a chronoloqical record of the 
course of the project over the three years of its life. 

Tho technical papers that have been prepared arc listed in 
Table 1. Most of these papers were prepared for the purpose of 
eventual publication. At the time of this writing, three of 
these papers have been published either in a professional journal 
or in a conference proceedings, or both; five have been submitted 
to journals for consideration for publication; and three more 
will be submitted shortly. 

In addition to progress reports and technical papers, three 
other documents have been prepared. These are: (a) a set of 
narrative reports on the activities and achievements of individual 
students, prepared by the teachers, (b) an operators reference manual 
and (c) a draft curriculum manual. These documents are listed 
as numbers 12, 13 and 14 in Table 1. 

Each of the documents listed in Table 1 should be considered 
as part of the final report on this project. A complete set of 
these documents is being forwarded to BEH; copies of individual 
reports are available on request. There were two reasons for 
preparing a number of separate documents rather than attempting to 
integrate everything in one report: (a) to produce some manuscripts 
that would be appropriate for submission to technical journals, and 
(b) to facilitate selective distribution of the recults of this 
project to other researchers and/or prospective users of this, or 
a similar system. * 
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Table 1. List of documents prepared under 
Contract No. OEC-O.l-71-4 67 0 



1. Niokcrson, R, S. & Stevens, K. N. An experimental computer- 
based system of speech training aids for the deaf. Proceed j ngSy 
Conferonce on P pe ecli Cor-nu n ica 1 j o n a n d F roce ssincT^, I EEj r ^ nd~ 
Air Force CambTTdcje kcsearch Laborat'ofTea , April", 197~2/ 238-241. 

2. Nickerson, R. S. & Stevens, K. N. Teaching speech to the donf: 
Can a computer help? P roccc <l - '.i , Associat ion f or Computing 
Macli ery , August, 1972, 240* Ij a nd ' IhfEL Tra n sac t ion's" o n ~' 
Aud io "anc'i Electroacoustics , L. /3, AU-2'1 , 4^45-455. 

3. Nickerson, R. S. , Kalikow, D. N. , & Stevens, K. N. A computer- 
based system of speech-training aids for the deaf: A progress 
report. BBN Report No. 2901, i'eptcmbor, 1974. Abbreviated 
version published in AF TPS Conf ere nce Proceedin gs, 1974, 43 , 
125-126. Also submitted to Ap.eric an Annals of tKe Dea f. 

4. Stevens, K. N. , Kalikow, D. N. , & Willemain, T. R. The use of 
a miniature accelerometer for detecting glottal waveforms and 
nasality. BBN Report No. 2907, September, 1974. Accepted for 
publication. Journal of Speech and Hearing Research . 

5. Stevens, K. N. , Nickerson, R. R. , Boothroyd, A., & Rollins, A. 
Assessment of nasality in the speech of deaf children. BBN 
Report No. 2902, September, 1974. 

6. Nickerson, R. S. , Stevens, K. N. , Boothroyd, A., & Rollins, A. 
Some observations on timing in the speech of deaf and hearing 
speakers. BBN Report No. 2905, September, 1974. 

7. Stevens, K. N., Nickerson, R. S., Rollins, A., & Boothroyd, A. 
Use of a visual display of nasalization to facilitate training 
of velar control for deaf speakers. BBN Report No. 2899, 
September, 1974. 

8. Boothroyd, A., Nickerson, R. S. , & Stevens, K. N. Temporal 
patterns in the speech of the deaf — An experiment in remedial 
training. Report from the Research Department, Clarke School 
for the Deaf S.A.R.P. #15, September, 1974. 

9. Boothroyd, A., Archambault, P., Adams, R. E. , & Storm, R. D. 
Use of a computer-based system of speech analysis and display 
in a remedial speech program for deaf children. (Based on a 
panel presentation to the Massachusetts Speech and Hearing 
Association, May, 1974); Clarke School for the Deaf S.A.R.P. 
#14, September,' 1974. 



ERIC 



Report No. 2914 



Bolt Doranek and Newnian Inc. 



TablG 1 (continued) 



10. Boothroyd, A. Tcncher/riv-carchor intoraction — A model and an 
example. rrocoa d i nnr. (•*♦' t lio 4Gth MctM inq of Amcricvm 
In str uct.o r« of UKruoaf , T^'I'A , ' 122^ \ 2^, "u.s". Printing' Office, 
Washimi'ton, D.C. 

11. Nickerson, R. S. Spoc-ch training ond speech reception aids 
for the doaf: A review. To appear ay a 13I3N report. 

12. Adams, R. E. & Storm, R. D. Narrative reports on students 
who received speech training with tlio system. I^eport from 
the Research Department, Clarke Sc]iool for the Deaf S.A.R.P. 
<fj6, September, 197 4. 



13. Rollins, A, M. , Kalikow, D. N., & Nickerson, R. S. Operators 
Reference Manual and Software Documentation. BBN Report No. 
2904, September, 1974. 

14, Nickerson, R, S,, Stevens, K, N. , Boothroyd, A., Adams, R, E. , 
& Storm, R. D, Draft curriculum manual: Review of speech 
problems of the deaf and preliminary guide to the use of the 
BBN system of speech training aids. BUN Report No. 2911, 
September, 197 4. 
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The body of the final report proper contiists primarily of 
overview and summary-typo information, and "pointers" to the 
other documentir. v.iitre more rtotailcd information regarding spocifi 
aspects of the project can be found. * Tho final se'^tion containr> 
a number of conclusions that we have drawn froin the experience 
and result? that have been obtained over the three years of the 
project, and inc»ker, several recommendations concerning dircctionr; 
that might be taken by future efforts to develop speech-training 
aids for the deaf. 
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3. SYSTKM DESCRIPTION 

The system har. boon described in detail in papei'G nnriiborod 
1, 2f and 3 in Table 1. Only a brief suimnary description will 
bo given here. 

As illuctratod in Fig. 1, speech information is acqnired via 
a voice microphone and a mini ture (BHN Model 501) accelcjomctcr . 
The accelcrometer is attached to the nose to acquire information 
concerning nasalisation, and to the throat to obtain voice funda- 
mental frequency. The output from the microphone is fed into a 
bank of 19 band-pass filters covering the frequency range from 
100 to 6560 Hz. Wliv^n the accelerometer is attached to the nose 
its output is fed into a "nasalization" detection circuit; when 
it is attached to the throat its output is fed into a pitch 
extractor. 

The output of each of these "preprocessing" components; — the 
nasalization circuit, the pitch extractor, and each of the 19 
filters — is sampled by the computer (a Digital Equipment Corpora- 
tion PDP/8F.) every 10 milliseconds. These samples are continuously 
stored in an area of the computer's memory that we refer to as a 
"ring buffer." The size of the ring buffer is limited by the 
amount of computer memory that is not required for other purposes. 
Typically, we have used a buffer large enough to hold a represen- 
tation of a speech sample of two seconds' duration. The data that 
are acquired at any given instant replace the oldest data in the 
buffer, so that the system always contains a record of the speech 
produced during the immediately preceding two seconds. By pressing 
a button on the control panel, the user can, at any time, capture 
the speech that is currently represented in the buffer. When the 
capture button is pressed, the recording of new speech data stops 
until another button is pressed to inculcate that it should be 
resumed . 
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The data that arc stored in the rang buffer provide the 
basis for a variety of diK}:»lnys. These displays can bo operated 
in real time, providing the user with a visual repro.<.;onta tion of 
specified ar;pec:ts of his speech (e.g., voicing, amplitude, funda- 
mental frequency, nasalization) ar. he speaks. They also can be 
frozen nnd/or replayed for inspoctJon after the fnct. 

Four different display programs have been developed. 'One 
presents the child with a game-like situation, permitting him to 
"score baskets" by successfully accomplishing certain vocal exer- 
cises. A second program represents certain speech parameters in 
the features of a cartoon face. A third provides information about 
the spectral properties of sounds by means of forms whose shapes 
depend upon the v/ay the acoustic energy is distributed over the 
voice- frequency range. A fourth program provides the user with 
the ability to display various speech parameters as functions of 
time. 

In addition to the display programs, a fifth program has been 
developed for the purpose of facilitating the analysis of recorded 
speech samples. This program permits the user to listen to a speech 
recording, to capture a segment of interest, to display certain 
parameters of the captured segment, and to obtain various objective 
measurements from the program on demand. 

For more details concerning the system the reader is referred 
to the reports mentioned above. 
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4. USER DOCUMnNTATTON 

The user documentation thnt has been prepared is of two 
types: (a) instructions concorning the mechanics of loading 
programs into the computer, setting prr-jram parameters, modifying 
programs, and so on, and (b) guidance and suggestions concerning 
the use of the displays in training sessions. This documentation 
is contained in two manuals: an Operators Reference Manual 
and a Draft Curriculum Manual (roj^orts numbered 13 and 14, 
respectively, in Tabl^.' 1) , 

The tentative nature of botii docnir.fnt.s, particularly the 
latter, is recognized. We are keenly aware of the fact that the 
guidance and suggestions that we have to offer concerning training 
are based on far too little experience to warrant being considered 
anything but tentative. The curriculum manual, therefore, is 
identified as a "draft" in order to emphasize the fact that vje 
view it as, hopefully, something upon which to build. We think 
that much has been learned as a result of using this system in 
speech training classes over the past two years. However, antici- 
pating the final section of this report, we may note that one of 
the main conclusions that we have drawn from that experience is 
that current speech processing and display-generation technology 
hold promise for the problem of teaching speech to the deaf, but 
a great deal more effort will be required by both researchers and 
teachers in order to learn to exploit this technology fully. Many 
of the most urgent and difficult problems appear to be less tech- 
nical than pedagogical. 
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5. USE OF SYSTEM AT THE CLAKKi: SCHOOL 

During the course of thio work, 42 students (24 in 1972-73, 
18 in 1973-74) at the Clarke School were given regular speech 
training U5:;ing the cystom of displays. For most of those students, 
this training was primarily on a one-to-one basis with a teacher. 
During the final months of the 1973-74 school year, six of the 42 
students also participated in unsupervised drill with the system. 

5.1 Sqlectiqn_qf _Stu^ 

The selection of students for participation in this program 
was made on the basis of several considerations: 

(a) The types of speech problems of the students. It was 
desired to work with several diverse types of speech problems, so 
that experience with these problems could be obtained and so that 
the effectiveness of training with the system could be assessed 
for different types of displays. Furthermore, some students had 
severe problems with a number of different aspects of their speech, 
whereas others had relatively minor difficulties with one or two 
speech attributes. In all cases, the speech problems of the stu- 
dents were such that it was expected that work with the system 
could be beneficial to their speech. Among those selected for 
training with the system were several students who had specific, 
obvious and severe speech problems which had not responded signi- 
ficantly to conventional training methods. 

(b) Ago and academic level of the students. The ages of the 
participating students ranged from 8 to 17. Work on an experimental 
basis was carried out for a short period of time during the summer 
of 1973 with two younger children, ages 4 and 6. Except for these 
two children, an effective lower age limit on participants v;as 
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imposed by the Lower School policy of not atlompting one-to-one 
analyt ic speocn v;ork with children in their first and second 
years of school. 

(c) Hear in q level. Of the 42 students in the age range 8-17, 
one was severely deaf, 8 were profoundly deaf, but with high- 
frequency residual hearing, 17 were profoundly deaf with only 
low-frequency residual hearing, and 17 were possibly totally deaf. 
The definitions of these categories of deafness are given in 
Table 2. 

(d) Availability of students f oJ^._gchedulinq . Students were 
scheduled for tutoring with the system during the time of their 
regular speech class. Thus, scheduling probl^^ms imposed some 
limitations on the selection of students for this program. 

(e) Individual requests by teachers and students . As the 
existence of the computer-based system and its capabilities 
became known in the school, occasional requests for tutoring with 
the system came from students and from teachers. Within the 
constraints described above, attempts were made to accommodate 
these requests. 

5 . 2 Ch^r a c t e r i s t ic s _o f _S tu d en t s 

The students were approximately equally divided botv;een girls 
(24) and hoys (18) . Host of the students (28) were tutored for a 
7-week period, ten were in the program for 14 weeks, and only four 
were involved for over 14 V7oeks. There were somewhat tnoro students 
in the 8- to 9-year-old group (15) and fewer in the 16- to 17-ycar- 
old group (4) than in the other age ranges. 
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Tabic 2. Definition of catccjories of hearincj loss 

severe: better ear three-frequency avorarje 

(500, 1000, 2000 Mz) of 90 dB I.S.O. or less. 



profound with high-frequency hearing: better ear 

three-frequency average in excess of 90 dB 
I.S.O. ; thresholds at 2 or 4 kHz of 110 dB 
or less. 



profound wi i:h low-frequency hearing: better ear 

three-frequency average in excess of 90 dB 
I.S.O.; thresholds at 2 and 4 kHz in excess of 
110 dB, but thresholds at 125, 250, or 500 Hz 
less than 60, 75, or 100 dB, respectively, 

profound, possibly total: better ear three-frequency 

average in excess of 90 dB I.S.O.; no measurable 
pure-tone responses, or responses are probably 
attributable to touch (i.e., in excess of 60, 
75, and 100 dB at 125, 250, and 500 Hz, 
respectively) . 
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Tablo 3 sli(n;:5 th.a nurhnrs of Ptudonts v;jio v;orked on Rpt^ciiic 
types of spoGch prohloms. Of the A2 students, 21 concontrntrd 
on only one probion, 16 vorked on t\\'0 problcps, and five workcrl 
on three problems. The table shov;s that the problems receivincj 
most attention v:ere tirrdno, intonation, and liypornasality , 

5 • 3 Diagnosis 

A crucial component of any speech training program is the 
diagnosis of the speech problems of particular students selected 
for training, and the formulation of training objectives for these 
students. During the first year of work with the system at the 
Clarke School, initial diagnosis was accomplished primarily on the 
basis of subjective assessment by the teachers — the classroom 
speech teachers, the speech coordinator of the school, and the 
research team involved in using the system to aid in speech training. 
Each student's speech problems were characterized in terms of 
several categories: timing problems; problems with pitch range and 
control of pitch; problems with velar control, leading to hypo- 
nasality, hypernasality , or both; problems with voice quality, par- 
ticularly breathiness or harshness; and problems with articulation. 
Problems with articulation were usually specified in greater detail, 
such as difficulty with /s/, inadequate vowel range, inadequate 
diphthongization, problems with consonant clusters (or blends) , etc. 

Following the initial diagnosis, recoirrocndations were made 
that a student receive training in one of these areas of emphasis. 
Initial work with the student in tutorial sessions with the displays 
usually confirmed the initial sul:>jective diagnosis, and from this 
evaluation it v;as possible to formulate short-rsmgo objectives 
for the training sessions. Some of the students selected for 
tutorial work in the first year wore those with very bad speech 
whose problems had long resisted reredlation. 



Report No, 



2914 



Bolt Beranck and Ncvmian Inc 



t5> 




C 




•H 




M 




0 




4J 








4J 




<U 


E= 


C 


0) 


o 


iH 


1 




0 


0 


4J 




1 




o 




c 


tt-i 


0 


o 


c 


0) 








>, 


x: 














o 




(0 




0) 












o 








•a 


to 








0 


u 




0 






o 






o 


a 


xi 






x: 




o 


(0 


<u 


4J 




C 


a 


0) 


(0 


'0 










x: 


10 








«H 


0) 


0 






o 






a 


(/) 


XI 




e 


<u 






c 








<u 


■p 


x; 






(A 




•H 




x: 


c 




id 



o 

rH 

o 

< 



4-» 



<I5 

cr 

a 
o 

O 
> 



00 
H 



(0 

o 

H 



n 



a f-< 

a: 



U 



§ 

O 



o 

CO 

U 



0) 



rH CO 
M-4 CO -H 
O •H (0 

crt o .C 

^4 3 E 

< H W 



CM 



CM 



•H 
M 

O 



CM 



CM 



ON 



c 
o 

CJ 
0) 



in 



.H 



H 



4) 



o 



(0 
O 



00 



cn 



CM 



CM 



iH 
O 

H 





* 












«> 




•p 








iH 








o 


* 


•H 






0) 




\ 






0 


C 


4-1 


d 










a 


(0 


c 












0 










x: 


4J 


a. 


c 


•H 




no 










5 


(0 


(0 


iH 




<D 


w 






% 




> 






•It 




•K 



14 



0 



Bolt Boranek and Nevmian Inc 



During the Kocond year of work with the system at the school, 
diagnor>is of speech problems and formulation of training objectives 
were carried out on a more systematic and quantitative basis. 
Each student who was a potential candidate for work with the system 
was given an initial evaluation based on objective ineasuremGnts 
and on subjective assessments for a specified body of speech 
material generated by the student. Some of tl)e data v^ere collected 
on-line by making measurements on the displays with the student 
present; other measurements and judgments were made subsequent to 
the initial recording session. Tliis evaluation consisted of the 
following : 

(a) Measurements of average fundamental frequency (F^) and 
Fq range for a selected sentence. 

(b) Measurements of HL reading (see Report #3 of Table 1) . 

A high reading on this scale is a tentative indication of breathy 
voice quality. 

(c) Comparison of "nasalization readingr;" for nonnasal vov:cls 
and nasals in selected one- and tv;o-syllaljle v;ords. 

(d) Measurements of durations of selected speech events in a 
simple paragraph read by the student. 

(e) Subjective judgments by the two research teachers of 
various speech attributes relating to voice quality, pitch, and 
timing, based on listening to the recorded paragraph. 

(f) Subjective judgments of the articulation of selected 
vowels and consonants occurring in a list of words recorded by 
the student. 

On the basis of all of these data, a "dossier," describing 
the speech of each student, was prepared. This included also 
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information on tho student's audiogram and intolliqibility scorns 
achieved in the regular intelligibility testing program at the 
school. One of the items in this dossier was the "speech profile," 
shown in Fig. 2. Problems with articulation are assessed by 
critical listening to the items in tho v;ord list; problems with 
voice cjuality, pitch, and timing are determined by rating of the 
various attributes indicated at the left of the figure on a 5- 
point sc^le. The percent intelligibility (from the standard 
Clarke School tests) is noted at the bottom of the profile. Nota- 
tions concerning average fundamental frequency, range of fundamental 
trequency and syllabic rate during the sentences are made to the 
right of the chart. 

The data in Fig. 2 and the objective measurements listed 
above were supplemented, in the case of some students who were 
diagnosed to have specific speech problems, by additional objective 
measurements of timing, nasality, and HL value. 

This material was used to provide a preliminary diagnosis 
of the speech problems of students selected for inclusion in the 
speech-training program during the second year at Clarke School, 
and to provide a basis for the establishment of initial training 
objectives. In some cases, the early training sessions revealed 
more detailed speech problems not uncovered in the initial 
material used for diagnosis. 

In addition to being used to help diagnose the speech of the 
children who were to participate in the project, the system was 
occasionally used, at the request of a teacher, to provide a 
quantitative estimate of some property of the speech of a child 
(e.g., the fundamental frequency, the degree of nasalization, 
etc.) who was not participating in the program. 
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Fig. 2. Method of summarizing data from subjective evaluation 
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include data from objective measurements of fundamental 
frequency and timing. 
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Over the course? of this project wg have become incrcaKinyly 
aware of the importance of diagnosis in the overall speech-training 
program. A detailed diagnosis of an individual child's speech 
deficiencies is a prerequisite to the establishment of reaeonablo 
training objectives, and of training procedures that are likely 
to realize those objectives. It is also essential to the accurate 
assessments of progress that a child may be making as a. result of 
training. We have expressed our views on this subject in Section 
3 of the Draft Curriculum Manual. 

It should be emphasized that the diagnostic procedure that 
is described above is only a tentative one, and in need of con- 
siderable refinement. It represents a step, however, in the di- 
rection of producing the type of quantitative information that will 
be necessary, we think, for truly effective training procedures. 
We believe that the development of a comprehensive, quantitative, 
easily-administered diagnostic procedure could represent a major 
advance on the problem of speech training for the deaf. If such 
a procedure could be developed, one can imagine that it might be- 
come as important a tool for education of the deaf as audiomotric 
testing procedures currently are. A speech-proficiency profile 
could, in principle, represent for speech what the audiogram 
represents for hearing. 

A second and extremely important potential benefit that would 
result from the widespread use of a standard speech-diagnosis pro- 
cedure would be the accumulation over time of a comprehensive body 
of data that could be used to describe, in objective and quantita- 
tive terms, the characteristics of the speech of the deaf. 
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5.4 Tutorial Instruction 

The instruction of the students in this program consisted 
primarily of tutoring on a ono-to-one banis with both student and 
teacher in front of the displays. In the eecond year of the pro- 
gram, a few students were involved in ursupervised drill. The 
"self-instruction" node will he describod in the following section 
of the report. 

Almost all of the tutorial instruction was carried out by 
two teachers. These individuals (Mr. Robb Adams and Mr. Robert 
Storm) were qualified teachors of the deaf at the Clarke School, 
and each was assigned half-time to this project. The remainder of 
their time v;as spent in classroom teaching. Before tutorial ses- 
sions with the system began at the school, these teachers received 
training in the use of the system and in some principles of acoustics 
and phonetics relevant to proper interpretation of the displays. 

Some tutoring v;ith the system was also carried out by the 
Director of Research at the school (Dr. Arthur Booth royd) , and, 
on occasion, by student teachers. 

The training sessions consisted of various types of exerciser 
that were graded in difficulty. The general approach was to work 
with ths student until he could achieve a reasonable level of 
performance in one type of exercise before proceeding to the next. 
The teachers used some judgment in selecting tasks for students, 
however, in the interest of maintaining motivation in the students 
and avoiding long sessions in which they failed to achieve suc- 
cesses. (Training procedures are discussed in greater detail in 
documents 7, 8, 9 and 14 in the list of Table 1.) 
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The types of cMcrcisos prepared for the students depended, of 
course, on the ciBi-ect of speech that was being considered. For the 
most part, however, those exercises wore of two general typoi: 

(a) Work with "vocal gymnastica, " in which the sounds produced 
by the student are nonspocch activities such as sustained vowels 

qr consonants, simple nonsense syllables, or sequences of such 
syllables, 

(b) Production of meaningful utterances of various lengths, 
selected to contain the speech gesture that is being worked on. 

For the vocal gymnastics activities, it was possible in some 
situations to set objective criteria that the student v/as asJied to 
achieve, such as producing an isolated vowel with nasalization 
reading below a specified value, or producing a sustained vowel 
with pitch in a specified range, or with a particular terminal 
fall in pitch. In other cases, the teacher generated a "target" 
pattern, and the student was required to match this pattern, the 
judgment of adequacy of match sometimes being a subjective one. 
Examples are the timing of a sequence of two nonsense syllables 
such as pa pa, or the generation of /s/ with the proper vertical 
spectrum pattern. Suggested sequences of training exercises for 
specific classes of speech problems are given in the Draft Curri- 
culum Manual. 

Initial orientation and training were, of course, carried out 
with the displays being generated in real time, yielding an in- 
stantaneous record of the student's speech activities. After this 
initial training, however, some exercises were carried out with 
the system in "delay mode," such that the student did not observe 
the display as he was producing the utterance. In this case, the 
display was put on the screen only after the student had had an 
opportunity to assess the adequacy of his utterance. This component 
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of the training activity was carried out in an attcrpt to "woan" 
the student away frcn reliance on the disj^lay and to accelerate 
the process of internalization of tiie student's skills and carry- 
over to spontaneous speech. Thin internalization process was em- 
phasized only during the second year of use of the system at the 
Clarke School. 

5.5 Unsupervised Prill 

Although we did not originally plan for this particular system 
to bo used by students without a teacher present, it became apparent 
during the course of the project that many of the students were 
quite capable of using the system in this way. Therefore, toward 
the latter part of this year v:e began experimenting with this 
possibility. 

Six children, between the ages of 12 and 17, have used the 
system in this V7ay. Each had participated in the tutoring sessions 
before using the system in "self-instruction" mode. The average 
numbers of tutorial and unsupervised drill sessions for these 
children were 43 and 36, respectively. In some cases, the two 
types of sessions were interspersed. 

During a self-instructional session the child is on his ovm. 
He comes to the "orange room" (the name given to the room whore 
instruction with the computer displays is given) , turns on the 
computer, puts on the microphone and accele remoter, selects the 
display with which he is to work, practices exercises that he and 
a teacher have agreed upon, and then shuts the computer off and 
leaves, when he is done. Simple self-scoring procedures have been 
developed and are used by the children during these sessions to 
record their progress as they work toward specific objectives. The 
children seem to enjoy working with tlje system on their own; there 
certainly has been no evidence that they are intimidated by the 
fact that they are interacting with a computer. 
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Using thG system in a self-instructional mode has several ad- 
vantages, perhaps the most obvious of v;hich is the fact that it 
does not require the expenditure of the teacher's time. It also 
has a danger, however. Because a teacher is not present and 
monitoring the performance of the child, there is the possibility 
that while he is working on one aspect of speech, other aspects, 
to which he is not attending, may be affected negatively. It is 
important, therefore, at least until adequate self-instructional 
procedures are developed, that the effects of self-instruction be 
checked periodically by teachers in order to insure that bad speech 
habits are not being practiced inadvertently. We do feel, however, 
on the basis of our experience to date, that self-instructional 
use of such systems holds very real promise. VJe suspect that for 
many children the optimal training strategy will involve some 
mixture of tutorial and self -instructional training, 

5.6 Normative Data Collection 

A final purpose for which the system has been used is to 
collect data that could be helpful in establishing quantitative 
training objectives with respect to certain aspects of speech. 
If one is to attempt to improve the speech of a deaf chi3.d with 
respect to some particular property or properties, : t is necessary 
to know not only how the speech is deficient but what must be done 
to make it correct. Unfortunately, it often is not possible to 
determine from existing data exactly what one should work toward 
in the case of many properties of speech. We have, therefore, in 
S€!me instances, collected data from normally-hearing speakers to 
help provide training objectives with respect to some speech 
properties. In particular, we have done this with respect to 
timing and nasalization. The results of these efforts are de- 
scribed in papers numbered 5 and 6 in Table 1. The use of these 
data in the establishment of training objectives and for the 
purpose of evaluating effects of training has also been documented 
in papers numbered 7 and 8 in Table 1, 
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6. EVALUATION OF EFFECTIVr.NESS OF 2PF.ECH TRAINING USING THE 
SYSTEM OF DISPIJVYS 

The effectiveness of the speech-training program was assessed 
in a number of different ways. These include: 

(a) Daily and weekly records maintained by the teachers of 
the objectives and accomplishments of the children for the indi^ 
vidnal training sessions. 

(b) Subjective evaluation, by each teacher, of the overall 
progress of his students over the course of their entire training 
programs. 

(c) Objective data on relevant acoustic characteristics of 
test utterances recorded before, during, and after, training. 

(d) Subjective judgments on the utterances noted in (c) , 
including judgments with respect to particular attributes of the 
speech (timing, nasality, etc.). 

(e) Measures of speech intelligibility. 

6 . 1 Tra inin g Session Documentation 

As v;e have already noted, the speech of each child who par- 
ticipated in this project was diagnosed before training was begun. 
On the basis of this diagnosis the long-range goals for the child 
were established. These goals were relatively general, e.g., to 
improve the speech with respect to some particular aspect or 
aspects. 
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Specific daily objectives, consistent with the long-range 
goals, wore set for each child on a wc?ok-by-weok basis. These 
objectives were of the sort: 

- Using short sentencGB,to articulate unaccented syllables 

at or slightly below uis habitual pitch, while making accented 
syllables at least 50 Hz above his habitual pitch (for a child 
with a habitual pitch of 280 Hz), with the terminal fall being 
the lowest pitch for the utterance. 

- To produce utterances with nonnasal consonants with a 

nasalization rerdinq of under 50, and nasal consonants with 
a nasalization reading greater than 200. 

- To produce two- and three-syllable utterances with different 
stress patterns, making the duration of the stressed syllable 
three to four times the duration of the unstressed syllable 
when the stressed syllable appears in final position, and 

one to two times as long when it appears in other than final 
position. 

In order to document the progress of each child vis-a-vis 
these objectives, the teachers maintained daily records in which 
they described the exercises that were used during each training 
session, and indicated the degree of success realized by the 
child during that session. On occasion, these data were re- 
corded systematically in such a way that an assessment could be 
made of the percent of trials that were successful, the number of 
trials needed before some criterion performance was achieved (such 
as five successive correct attempts) , or the percent of trials 
in which the student was able to assess correctly the adequacy of 
his production (with the display in delay mode) . During the latter 
part of the program, occasional snapshots of the display showing 
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the student's performance with particular utterances were made to 
supplement the written records. 

A v;eekly summary of the information contained in the daily 
records was also prepared by each tearber for each of the chi Idren 
with which he w<ir> working. These summaries were reviewed and 
discussed by the project team during the meetings that were 
regularly held for the purpose of assessing the overall progress 
of the project and to revise training objectives for the individual 
students as progress warranted. An example of the e summaries is 
shown in Fig. 3. 

Both the daily records and *-he weekly summaries contain many 
data that are germane to the question of the effectiveness of 
this approach to training. This documentation demonstrates 
clearly that all of the students made some progress in the train- 
ing sessions, in the sense that they were able to accomplish speech 
exercises or produce utterances for which they were not successful 
at the onset of training. Examples of data that show this type 
of progress are shown in Figs. 4 and 5. Figure 4, for example, 
demonstrates that a student who was initially unable to produce 
a sentence without nasalization was able after some training to 
correct this problem and consistently to generate the sentence 
with a raised velum. Figure 5 gives the pitch contour of a par- 
ticular utterance with an inappropriate terminal rise before 
training and with a well-executed terminal fall after a period of 
training. 

At the end of a student's training period (7 or 14 weeks for 
most students) , a final summary was prepared for that student. 
The daily and weekly summaries were used as input material for 
this purpose. These summaries comprise the report listed as 
number 12 in Table 1. 
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Student's N, 



A.M. 



Tutor 



Robb Adams 



Date (3) 



M arch 8-14 , 197± 



Week // 10 (5 sessions) 



Emphasis: velar control and timing 



Obj ective s; 

To produce nonsense syllables and words with appropriate timing features. 
(Ratio ol duration of unstressed syllables to duration of stressed syllables 
is approxima t<^ly 1/3 to 1/A. Intranhrase gap is approximately 0 to 1/3 
times the unstressed syllable.) 

To produce utterances (nonnasal) under a nasality level of 50. 

To produce utterances with appropriate timing and nasality features. 



A.M. produced utterJmces with appropriate timing and nasality features. 
Specific results are attached. 

The student is inconsistent with his velar control. One day he will succeed 
with most utterances attempted and the next day he will have difficulty with 
most utterances attempted. Specific results are attached. 

Timing and velar control were evaluated with the VN display in flow mode. 
Only a few utterances were attempted. Results are attached. 

Results and Comments : 

A.M. is more successful with nonsense syllables than real words because of 
the articulation requirements involved. It is a slow process working from 
nonsense to meaningful utterances that he can succeed with. 

Success with velar control is sporadic, even with nonsense material. Certain 
sounds are usually nasal for the student (/oo/, /ee, /a-e/, /o-e/. /i-e/. 
/I/, /u-e/, /d/). 

Becausr of limited exposure it is difficult to say at this time how effective 
the VN display will be for evaluating timing as well as nasality. The student 
needs to work until it is easy for him to articulate one-, two-, and maybe 
three-syllable utterances initially. Timing alone is not very difficult for 
A.M., but as the length or rate of an utterance increases, velar control and 
articulation fall apart. 

Subsequent Activities ; 

Produce utterances with appropriate* nasality and timing features. 

^Description of appropriate timing and nasality features appears under 
Objectives above. 

Pig. 3. Weekly summary sheet - computer/speech program 



Progress ; 
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Fig. 4. Performance of a deaf student in producing the sentence 
"I live at Clarke School" with nasalization reading 
(output of accelerometer attached to nose) below a 
specified value. The ordinate is the number of trials 
necessary before the first of 5 consecutive successes 
was achieved. 
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Fig. 5. The photoqraph at the top shows the presence of a sudden 
pitch rise at the termination of the utterance "Audrey." 
This inappropriate behavior was characteristic of this 
student's speech. The bottom picture shows thAt, after 
some training, the student eliminated the pitch jump and 
produced a natural terminal fall in pitch. 
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6.2 Subjective Teacher Asscg;smcnt of Ovorall rroqrcss of 

Following completion of each student's participation in the 
project, his performance v;as rated by his tutor v;ith respect to 
four aspects: 

a. improvement of vocal gymnastic skills 

b. incorporation of these skills into rehearsed speech 

c. carryover to unrehearsed speech (formally presented) 

d. carryover to spontaneous speech 

Ratings were made on a 5-point scale as follows: 

0. no improvement noted 

1. a small improvement noted, but of questionable significance 

2. a moderate and significant improvement noted 

3. considerable improvement noted, but student failed to 
meet all criteria or showed inconsistent performance 

4. student showed considerable improvement and learned to 
meet criteria consistently and with ease 

Ratings were assigned on the basis of the goals that were set 
for tutoring. Thus, two students who received the same rating were 
not necessarily accomplishing tasks of the same difficulty. 

The results of the ratings have been discussed in the 
paper numbered 9 in Table 1. Table 4 shows the ratings of 
the students on each of the four dimensions for the speech 
area of primary emphasis. A comparison of ratings on the four 
dimensions shows that the dimensions of vocal gymnastics and 
rehearsed speech were rated significantly higher than the two 
dimensions related to carryover. This is in keeping with our 
observations that the learning of gymnastic skills and their 
incorporation into rehearsed speech could generally be accomplished 
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with little difficulty in a 7-weck period. Tho generalization of 
these skills to unreihcarsod and Hpontant^ous cpcoch required much 
longer periods of tutoring and even then was accomplished by only 
a small number of students. 

Further analysis of the data revealed tendencies for girls to 
be rated higher than boys, for older students to be rated higher 
than younger students, and for students with more residual hearing 
to be rated higher than students with less residual hearing. These 
were only trends, however. None of the differences reached the 51. 
level of significance. 

Significant differences were demonstrated between the ratings 
for students participating in Years 1 and 2 of this project, with 
those on Year 2 scoring higher. It will be recalled that the 
students in Year 1 tended to be selected from a pool of "difficult" 
cases, whereas a more representative sample was chosen for Year 2. 
It should also be pointed out that much more attention was given 
to internalization and carryover in Year 2 and it was only on the 
dimension of carryover to spontaneous speech that the difference 
between Year 1 and Year 2 ratings reached the 5% level of signifi- 
cance. A further factor or relevance to this observation is the 
increased experience of the tutors during Year 2 of the project. 

A comparison of the students on the basis of speech problems 
revealed few differences between pitch, nasality and timing groups. 
The pitch group was, however, rated significantly lower on the 
dimension of carryover to spontaneous speech (difference signifi- 
cant at the 5% level) . 

^ • ^ Objective Data on Relevant Attribu tes of Tcst_Utteranccs 

Before training was commenced, a series of utterances was 
recorded for each student involved in ^he training program. The 
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recordings were made on two channelS"-one channel was the micro- 
phone sicfnal, and the other was the signal from the acceloromotcr 
attached to the nost». The recorded material consisted of: (a) 
a list of 33 monocyllabic words, each elicited from the students 
by showing a picture; (b) a series of short phiases and sentences; 

(c) an approximation to spontaneous speech obtained by having the 
student tell the story described by a sequence of pictures; and 

(d) a list of six 10-word sentences drawn from a larger list that 
is used routinely to evaluate the intelligibility of the speech of 
students at the Clarke School, The same set of material vjas 
recorded within a few days after speech training with the system 
was terminated. These recordings were available for making ob- 
jective measurements of various attributes of the utterances of 
the students. The recordings could also be used to obtain sub- 
jective judgments of different aspects of the speech, and to assess 
the intelligibility of the speech before and after training. 

The pre- and post-training recordings just described were 
occasionally supplemented by recordings of special speech material 
that was designed to assess the progress of students with regard 
to particular speech attributes. For example, the reading of a 
short paragraph was recorded for the purpose of assessing certain 
aspects of timing. Some of these recordings have been analyzed 
or evaluated for the purpose of specifying the properties of the 
speech of deaf students in comparison with normally-hearing 
individuals, in order to provide a rational basis for the diagnosis 
of the deaf students. The results of some of these analyses are 
presented in reports listed as numbers 5 and 6 in Table 1. 

Comparisons between the before- and after-training recordings 
and the special speech samples recorded at various times during 
training were made for the purpose of assessing the effects of 
training on the speech of the deaf students. The results of some 
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of these compari'jons are pre.qcntod in reports listed in Table 
1 as numbers 7, B ond 9. 

Detailed datn of this kind were obtained from eight students 
who received training in tining, nine who were trained in velar 
control, and one whose tutorial sessions were concerned with 
pitch control. The data for timing showed that after some train- 
ing students achieved significant reductions in the duration of 
unstros.sed syllables and wi thin-phrase gaps, relative to the 
durations for the pre-training recording (report No. 8). Most 
students who received training with velar coiitrol were able to 
produce more nonnasal words with a raised velum after training 
than before — both words in isolation and words in sentences — 
although not all students showed consistent improvement for words 
in both types of utte^-ances (report No. 7) . The student for whom 
Fq measurements were made demonstrated an ability to eliminate 
inadvertent pitch jumps at the ends of v;ords and to prodvice an 
appropriate terminal fall in pitch (report No. 9). In summary, 
then, the available results demonstrate clearly that training was 
effective in modifying the speech of these students, making it 
more similar to that of hearing speakers with respect to certain 
objective measures of specific aspects, 

6« 4 Subjective Judgments of Test Utterances 

In cid-^ition to being subjected to a variety of objective 
measurements, some of the before- and after-training speech samples 
and special samples produced at different times during training 
were also used in tests in which listeners were asked to evaluate 
utterances in terms of specific properties. In particular, ratings 
of timing and rhythm, and of velar control, were obtained. The 
results of these tests are presented in reports listed as numbers 
7 and 8 in Table 1, Again, the results indicate improvements in 
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the speech of the participating students with respect to specific 
aspects, insofar as the quality of these aspects can be judged by 
listeners. The judqments of timing showed significant improvoment , 
whereas the judgments of velar control/ while demonstrating some 
improvement for some students, showed considerable variability 
between listeners. 

6.5 Measures of Intelligibility 

The ultimate goal of speech training is, of course, to increase 
the intelligibility and improve the quality of the student's speech. 
Whether it is reasonable to expect to make noticeable improvements 
in intelligibility as a result of 15 to 20 minutes of training a day 
over a period of a few weeks is questionable. Nevertheless, intel- 
ligibility measures were obtained for before- and after-training 
speech samples of several of the students who participated in this 
project. The results of these comparisons are given in reports 
numbered 7 and 8 in Table 1. The results were not consistent from 
student to student, and such gains in intelligibility as were 
found were small. The measure that was used — number of words 
correctly perceived — was relative] y crude, however, and because of 
its sensitivity to numerous exogenous variables, would be a reliable 
indicant of changes only if the increases were large. 
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7. CONCLUSIONS 

We present here the conclusions that we have drawn as a result 
of our involvement in this project. Some of these conclusions 
follow directly from the data that are presented in the various 
project reports. Others are more in the nature of opinions that 
have been shaped by the combined effects of several factors, e.g., 
extensive review of the literature, observations of speech-training 
activities (both with and without the system) , conferences with 
teachers of the deaf, and interactions with deaf children. 

a. The project has demonstrated the feasibility of a close 
interaction between researchers and teachers in the development 
of innovative speech-training aids and procedures for their use. 
We do not ish to suggest that this is the first time researchers 
and teachers have interacted effe-^tively in this way. It does 
seem to be the case, however, that such collaborations are more 
often preached than practiced. Moreover, we have become increa- 
singly convinced of the desirability, if not the necessity, of 
such an approach. 

b. Students within the age range of those who participated 
in this study (8 to 17 years) have little, if any, difficulty in 
interpreting displays such as those that were used in this project. 
In par.ticular, the differences between acceptable and unacceptable 
patterns were readily learned. 

c. Some evidence was obtained (experience with two children, 
ages 4 and 6, during the summer of 1973) that some of these dis- 
plays might be used to advantage in the teaching of much younger 
children. 
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d. Many teenage students are capable of using such a system 

by themselves in a self- instructional mode. Such self-instructional 
use probably should be attempted, however, only after the student 
has had some experience in the more conventional tutored situation. 
Moreover, when a student does use the system by himself, his per- 
formance should probably be monitored periodically by a teacher 
in order to assure that the practice that he is getting is having 
the desired effects. For many children an optimal training program 
may include a mix of tutored and self -instructional sessions. 

e. The approach to speech training that was taken in this 
project appeared to have some motivational advantages. In general, 
the children worked willingly during the training sessions. Also, 
several requests were received from students who had participated 
at some time during the project to return for additional tutoring, 
or for use of the system for self instruction, 

f. The displays and training procedures that were developed 
proved to be effective in helping the students to attain a variety 
of specific training objectives. These objectives had to do with 
improving speech with respect to certain specific features that 
had been diagnosed as being defective. The following examples 
Illustrate the types of speech problems for which some degree of 
success was obtained: 

Timing^. With the help of displays of temporal patterns of 
speech, students were trained to produce phrases and sentences 
with improved timing, including a reduction in pauses, a shorten- 
ing of the duration of unstressed vowels and an increase in 
syllable-production rate. 
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Velar control. With the help of a display of nasalization 
(based on the nniplitudo of vibration on the si^rface of the none) , 
students were trained to produce vowels with reduced nasalization 
and to gain more appropriate temporal control of the velum in 
words and phrases. 

Pitch contr ol. Training with the assistance of displays of 
fundamental-frequency contours was effective in increasing the 
ability of some students to speak with the pitch in a more appro- 
priate range, to produce a fall in pitch at the end of An utter- 
ance, or to reduce the incidence of inadvertent pitch jumps. 

Breathiness . A display that indicated excessive low-frequency 
energy for vowels, and, by inference, an inappropriately spread 
posture of the glottis, was used with two students to improve 
their ability to produce vowels without excessive breathiness. 

Articulation . Displays of spectral distribution were used 
effectively with some students to improve the pronunciation of 
some consonants (in particular /s/ and /s/) and of certain vowels. 

For all of these aspects of speech, improvement in speech 
skills was most marked for training exercises and utterances that 
were rehearsed, and was less (and sometimes quite small) for 
unrehearsed meterial and for spontaneous speech. There were 
large differences in the improvements achieved by different 
i^tudents . 

Inasmuch as paired-comparison control group experimentation 
was not attempted, the data do not warrant the conclusion that 
improvements such as those that were obtained could not be 
realized as readily without the use of these displays. There is 
abundant evidence in the literature, ^however, that conventional 
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speech-training techniques have not been very successful. There- 
fore, evidence that a procedure works in the sense that it results 
in measurable improvements in speech can be taken as at least 
tentative evidence that the procedure is effective relative to 
others that have been used. 

g. The teacher-users of the system learned a considerable 
amount about speech by virtue of being exposed daily to visual 
representations of various aspects of both normal and defective 
speech. In particular, much of value was learned concerning 
specific problems in the speech of the deaf and aL^ut the effective- 
ness of various approaches to training resigned to alleviate 

those problems. We believe that such a system could play a very 
useful role in teacher-training programs. 

h. The system proved to be helpful in the quantitative 
diagnosis of specific speech problems and in the objective 
assessment of progress that individual students were making with 
respect to these problems. Because of this more quantitative 
approach to the assessment of the speech of the students and to 
the evaluation of the progress of students, it was possible to 
be more precise than is typically the case in the specification 
of a graded sequence of exercises and training tasks. 

i. There is a need, however, for the development of more 
thorough and more quantitative procedures for diagnosing the speech 
deficiencies of individual children. The criticalness of this 
need became increasingly apparent to us over the course of the 

i "oject. 

j. There is also a need for more complete normative data to 
provide the basis for the establishment of specific training ob- 
jectives, given specific deficiencies, 
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k. A third need is for more intensive research aimed at 
the determination of the relationship between the objective and 
perceptual properties of speech, and especially the speech of 
the deaf. In particular, there is a critical need to know how 
the objectively measurable properties of speech relate to its 
intelligibility and perceived quality. 
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8. REC0M14ENDATI0NS 

We believe that the philosophy that has guided this project — 
that of involving researchers and teachers in a collaborative 
effort to evolve an effective system of speech-training aids — has 
proven to be a sound one. The system that has resulted from this 
effort has been applied effectively to the speech-teaching task. 
And, although the system has continued to evolve over the course 
of the project, its potential, we believe, has not come close to 
being fully realized. Many ideas about further display possibi- 
lities have not been explored simply because of resource limiLations 
What has been learned, however, can provide a solid basis for the 
furtherance of this approach. Moreover, the system remains highly 
flexible emd the programming can be readily modified so as to 
provide a vehicle for the testing of new ideas. 

The next step that should be taken, we believe, is to involve 
a larger and more heterogeneous group of researchers, teachers, 
and deaf students in a coordinated effort to extend this approach. 
A wider exposure of the system to potential users is probably the 
best way not only to facilitate the further development of the 
system's capabilities but also to develop new methods that will 
make the best use of the capabilities that the system provides. 
Enough has been learned about the potential of the system and 
about procedures for using it to warrant the installation of 
systems similar to the one that has been used at Clarke at a small 
number (say, three or four) of other schools for the deaf, and to 
engage the users in a collaborative effort to extend this approach. 
We emphasize "collaborative" because we feel strongly that the 
establishment of a consortium with a common goal, common tools, 
and a continuing interchange of ideas and information is the way 
tp maximize the chances of significant progress in this area. 
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Finally, there is a need for the development of training 
procedures that are designed to take advantage of the speech- 
processinq and display-generation capabilities that modern 
technology provides. We have become increasingly convinced that 
the most serious impediments to the development and effective 
use of instrumental training aids are pedagogical uncertainties 
rather than technical limiLations. There now exists a technology 
for processing speech and generating real-time displays that 
cannot be fully exploited until more is known about hov? speech 
should be taught. 
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